Video processing and rendering

ABSTRACT

Video processing apparatus in which input images of an input video sequence are processed according to user adjustable video effects parameters to generate output images of an output video sequence comprises: rendering means for rendering images of the output video sequence according to a priority order of: 
     (i) an image or images currently displayed for view by a user; 
     (ii) first and last images of the output video sequence; and 
     (iii) remaining images of the output video sequence.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to image processing and rendering.

2. Description of the Prior Art

In image processing systems such as video special effects systems, oncea particular effect has been selected and all the parameters defined, a“rendering” operation has to take place.

Rendering is the process of generating an output image, a series ofimages forming an output video sequence, or an intermediate data typesuch as a motion vector or position value, according to the processingoperation which has been set up. For example, a lighting effect mightinvolve the user selecting a source and a destination position for acomputer-generated spotlight to be applied to a video sequence. Oncethese positions have been defined, the next task is to render each imagein the output sequence by applying the defined lighting effect todetermine the colour and luminance of each pixel of each output image.

With current processing systems, rendering can be very time consuming,typically taking between a few seconds and many hours (depending on thesequence length) to render each image of a video sequence. To avoidinconvenient delays to a user, it is considered desirable not toinitiate a rendering process whenever a processing parameter is changedby a user, but instead to wait until the user is happy with all theparameters that have been changed. However, this does not give the usera direct response when he changes a processing parameter.

SUMMARY OF THE INVENTION

This invention provides video processing apparatus in which input imagesof an input video sequence are processed according to user adjustablevideo effects parameters to generate output images of an output videosequence, the apparatus comprising:

rendering means for rendering images of the output video sequenceaccording to a priority order of:

(i) an image or images currently displayed for view by a user;

(ii) first and last images of the output video sequence; and

(iii) remaining images of the output video sequence.

The invention addresses the problems described above by providing arendering operation which renders images or data values of an outputvideo sequence according to a priority order, so that those images whichare generally required most quickly by the user are rendered first. Thiscan alleviate the subjective delay involved in a rendering operation.This can alleviate subjective delays and also supports the use ofmultiple views of different stages of a rendering operation.

It is preferred that the user should be able to continue to operate theapparatus during a rendering operation, thus alleviating part of theproblem with previous apparatus (where a lengthy rendering operationcould make the apparatus temporarily unusable). It is thereforepreferred that the rendering means is operable as a background processwithin the apparatus.

In particular, this may lead to the user changing a parameter being usedby a current (background) rendering operation, so it is preferred thatthe apparatus comprises means for detecting a parameter change by a userwhich makes a current rendering operation invalid; and means, responsiveto such a detected parameter change, for aborting a current renderingoperation by the rendering means and for restarting operation of therendering means. This prevents a waste of processing resources inrendering using out-dated processing parameters.

Preferably the apparatus comprises user-operable sequence selectionmeans for selecting a subset of the input video sequence to be renderedto form the output video sequence. In this way the user can define arange of interest, with only those images in the range of interest beingrendered. Particularly during a development of the effects parameters tobe used, this can save rendering time.

The invention also provides a video processing method in which inputimages of an input video sequence are processed according to useradjustable video effects parameters to generate output images of anoutput video sequence, the method comprising the step of:

rendering images of the output video sequence according to a priorityorder of:

(i) an image or images currently displayed for view by a user;

(ii) first and last images of the output video sequence; and

(iii) remaining images of the output video sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the inventionwill be apparent from the following detailed description of illustrativeembodiments which is to be read in connection with the accompanyingdrawings, in which:

FIG. 1 schematically illustrates a digital video special effectsapparatus;

FIG. 2 schematically illustrates the organisation of operating softwareof the apparatus of FIG. 1;

FIG. 3 is a more detailed schematic illustration of the organisation ofoperating software for the apparatus of FIG. 1;

FIG. 4 schematically illustrates the propagation of an updated effectsparameter in the apparatus of FIG. 1;

FIG. 5 schematically illustrates the propagation of a re-render commandin the apparatus of FIG. 1;

FIG. 6 schematically illustrates a graph editor window and a palettewindow,

FIG. 7 schematically illustrates a graph editing operation;

FIG. 8 schematically illustrates the creation of a composite effecticon;

FIG. 9 schematically illustrates the file structure of a compositeeffect;

FIG. 10 schematically illustrates a viewer window;

FIG. 11 schematically illustrates an initial arrangement of theoperating software;

FIG. 12 schematically illustrates a previously proposed effects plug-in;

FIG. 13 schematically illustrates a new form of effects plug-in;

FIG. 14 schematically illustrates the relationship between the effectsservers and proxy effects for the effects plug-in of FIG. 13;

FIG. 15 schematically illustrates a system cache; and

FIG. 16 schematically illustrates a plug-in interface.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 schematically illustrates a digital video special effectsapparatus.

A digital video signal, comprising successive video images, is receivedvia an input interface 100 and stored in a disk array device 10. Thedisk array device 110 also stores any manipulated images generated bythe apparatus, and these can be supplied for output via an outputinterface 120.

A central processing unit 130 accesses data stored in the disk array tocarry out various video special effects in accordance with usercommands. The CPU 130 receives input from user input devices 140 such asa mouse and a keyboard, stores working data and programme code in amemory 150, and generates data for output on a display screen 160 via adisplay driver 170.

The apparatus can be implemented as a general purpose computer (e.g. aPC) running appropriate software, for example under the MicrosoftWindows NT® operating system. In the present embodiment, the disk arrayis connected to the CPU 130 via an UltraSCSI data link.

FIG. 2 schematically illustrates (at a very general level) theorganisation of operating software for the apparatus of FIG. 1.

The software is arranged as two categories of programme code: a coreframework, shown to the left-hand side of FIG. 2, and various“plug-ins”, shown to the right-hand side of FIG. 2. Once the softwarehas been initially loaded, the core framework is always present andcontrols parts of the operation of the apparatus which are sharedbetween different special effects processes. In contrast, the plug-insrelate to individual special effects (such as a lighting effect, amotion-tracking effect, and so on) and are loaded only when needed.

This is a very efficient arrangement because only those plug-insrelating to effects modules current required by the user need be loadedinto memory. This saves memory in comparison with a system in which aprogramme code for every possible special effect has to be loaded. Italso allows for a more rapid initialisation of the apparatus, avoidingthe need to load all of the program code into memory when the system isfirst started up, and can reduce any delays through code loading when anicon is first selected in the graph editor (see below). Furthermore, thearrangement allows a reduced subset of the apparatus (in particular, theoperating software) to be supplied or installed, containing just a grapheditor and the core processing but without the plug-ins. The system alsoallows third parties or users to produce their own plug-ins, so long asthey stick to a defined interface protocol between the plug-ins and thecore framework. So, a user could produce a bespoke effect very simply bywriting a relatively small plug-in programme.

The core framework and the plug-ins communicate with one another using aso-called “object linking and embedding” (OLE) protocol, described inthe book “Understanding ActiveX and OLE”, David Chappell, MicrosoftPress, 1996.

In the OLE system, a software designer can implement different sectionsof a computer programme as so-called “COM¹ objects”. Each COM objectsupports one or more COM “interfaces”, each including a number of“methods”. A method is a function or procedure to carry out a specificaction. COM methods can be called by software using that COM object. Thesystem is restricted so that other parts of the software using a COMobject can do so only via the defined interfaces—so they cannot directlyaccess programme code or data within the object other than via thedefined COM interfaces.

¹Component Object Model

So, in the present system, the core framework communicates with theplug-ins via these COM interfaces. (In fact, the core frameworkcomprises a number of intercommunicating objects capable of providing aCOM interface, but the basic principle in the same).

FIG. 3 illustrates the organisation of the operating software in muchgreater detail than that shown in FIG. 2. In FIG. 3, the diagram isdivided into four quadrants. An upper left quadrant shows so-calledviewer windows 310; an upper right quadrant shows effect user interfaces(UI) 320; a lower right quadrant shows effect “servers” 330 havingassociated parameter data (PD); and a lower left quadrant shows a coreprocessor 340 including a graph, along with an object manager 350, arender manager 352 and a change manager 358. At the interface betweenthe lower left and lower right quadrants is a meta database 354 whichforms part of the Windows NT Registry, and a palette 356 containingeffects icons, and default parameter values. The meta database 354 willbe described further with reference to FIG. 11, and the palette withreference to FIGS. 6 to 9 below.

Within the core processor 340 is a “graph”—in fact a “directed acyclicgraph” having a linked series of individual special effects. Each effectis represented in the graph as a proxy effect (PE) having an associatedcache (C) for storing the effect's output as soon as it becomesavailable, thereby allowing re-use of data from effects in a chain ofeffects if, for example, a parameter associated with an effect higher inthe chain is changed. Each proxy effect is associated with a respectiveeffects server 330.

The object manager 350 is responsible for controlling the lifetime andmemory management of active objects in the system. The render manager352 controls the starting, progress, prioritisation and stopping ofrender tasks. The change manager 358 controls undo/redo information andthe notification of changes to various parts of the system.

The basic division of FIG. 3 is that the two left-hand quadrants (upperleft and lower left) relate to features of the core framework, which arenot specific to any particular effect. These objects are loaded intomemory regardless of which particular special effects the user wishes toimplement. The two right-hand quadrants (upper right and lower right)relate to the plug-ins. Each plug-in has a server 330, which carries outthe processing associated with the effect performed by the that plug-in,and a user interface 320 which provides user interface controls (infact, for display in a viewer window 310) relating to that effect.

There is a similar top-bottom split in FIG. 3, whereby the two upperquadrants (upper left and upper right) relate to user interface controlsassociated with the special effects, and the two lower quadrants (lowerleft and lower right) relate to the processing and organisation carriedout to implement those effects.

Viewer windows give the user the opportunity to view the output of andcontrol parameters for a particular effect in a chain of effects builtup for execution by the apparatus. So, when a viewer window is opened bythe user, the output of an effect must either be generated or, ifavailable, retrieved from a cache store.

In a system of this type employing multiple COM objects, it is generallyconsidered undesirable that each object should save its working orresults data in a separate data file. Indeed, such an arrangement wouldgo against much of the reasoning which led to the establishment of theOLE system. Instead, data in this type of system is stored by all of theobjects in a single file or “compound document”, but with an orderedstructure within that compound document.

Basically, inside the compound document, a structure analogous to a fileand directory structure is provided. The equivalent of a directory is aso-called “storage”, and the analogy of a file is a so-called “stream”.Each compound document contains a root storage, below which is afamiliar tree structure of storages and streams. The COM streaminterface is simpler than the COM storage interface, but of course thestorage interface offers more flexibility.

In general, each COM object can be assigned either its own storage orits own stream in which to store its working data However, in ahierarchical system such as the present video special effects processor,where the core 340 co-ordinates operation of a number of plug-ins, thearrangement used in previous systems is to allocate a stream to eacheffects plug-in.

In contrast, in the present embodiment it is recognised that simplyallocating either a storage or a stream to a plug-in places too manyconstraints on the design and operation of a plug-in object. Instead, inthe interface definition between the core and the plug-ins, each plug-incan select between a stream, so that it can—if desired by the plug-indesigner—store its working data in a straightforward manner, and astorage, so that it can—if desired—store its working data under its own“directory” arrangement of streams and storages. This selection is madein advance by the plug-in designer creating the plug-in program code.

In the description which follows, communication protocols between thevarious objects will be described with relation to FIGS. 4 and 5. Theway in which the graph editor part of the core programme 340 is used toset up a chain of individual effects to form a composite effect will bedescribed with relation to FIGS. 6 to 9. The viewer windows and theirinteraction with the effect UIs 320 will then be described withreference to FIG. 10.

So, FIG. 4 schematically illustrates the propagation of an updatedeffects parameter in the arrangement of FIG. 3.

An example of this situation is that the user is setting up a “lighting”special effect, in which the effect of a light source is added to animage. The light source has user-definable source and destinationposition with respect to the image. If the user wishes to change one ofthese positions, this will invalidate any currently rendered output fromthat effect and also the change will need to be propagated between thevarious objects shown in the different quadrants of FIG. 3. This processis shown in FIG. 4.

Referring to FIG. 4, the user actually enters a changed parameter via aviewer window 310 (see FIG. 10 below). The viewer window is associatedwith a particular effect UI 320, which in turn is part of a plug-inrelating to an individual effect. (In fact, a single viewer window canbe associated with more than one effect, and not all effects need haveviewer windows open at any particular time, but the simplifiedarrangement of FIG. 3 will be maintained for this discussion). Theplug-in issues an “about to edit” notification to the core.

After the change has been made, the plug-in issues a “change occurred”message. This involves and/or initiates a series of actions. As a firststep 401, the viewer window communicates the updated parameter to theeffect UI 320. The effect UI 320 issues a “set” command to thecorresponding effect server 330, setting the revised value in aparameter store within the effect server 330. This is a step 402 on FIG.4.

In a step 403, the effect server 330 writes a “undo/redo” object intothe core 340, providing details of a handle or pointer which points to arecord (in the effects server) of the parameter before and after thechange. In response to receipt of this undo/redo object, in a step 404the core broadcasts a notification to all viewer windows that aparameter has changed and that some cached data outputs may beinvalidated. This notification is not specific to the viewer window inwhich the parameter change took place.

At a step 405, each viewer window issues a message to the correspondingeffect UI 320 requesting an update of its processing parameters. At astep 406, the effect UI 320 issues a “get” command to the correspondingeffect server to get the new parameter, and the new parameter isreturned to the effect UI 320 at a step 407. The effect UI thenpropagates the change for display in the viewer window at a step 408.

Generally, when a processing parameter is changed, this results in theneed to re-render the output of one or more effects. FIG. 5 illustratesthe way in which a re-render command is propagated through theapparatus, and follows on from the processing of FIG. 4.

At a step 502, the viewer window issues a re-render command to therender manager. The render manager then issues a re-render command tothe corresponding effects server 330. When the effects server hasfinished re-rendering the image, it issues a “finished” message to thecore 340. The core communicates this to the viewer window at a step 505,and at a step 506 and 507 the viewer window interacts with the effect UI320 to display the re-rendered effect output.

Where there are several viewer windows open, and several frames in arange of interest (which can be defined by the user as a subset—for testpurposes—of the overall video clip), images are rendered as multipleconcurrent tasks, according to the following priority order forallocating processing resources:

(i) an image or images currently displayed for view by a user;

(ii) first and last images of the output sequence of interest; and

(iii) remaining images of the output video sequence of interest.

As a further level of detail, before the render manager issues are-render command to the effects server, the render manager issues a“prepare to render” message specifying which image in the sequence is tobe rendered. The effects server responds with a notification of its“dependencies”, i.e. those rendered images which are essential beforethe request by the render manager can be executed. These might be imagesrendered by another effect (e.g. the or an immediately preceding effectin the directed acyclic graph) or images rendered by that effect itself.This latter case can occur in the example of a motion tracker, where inorder to render, say, image 5, the motion tracker needs its own renderedoutput for image 4.

In response to the messages received back from the effects server, therender manager sends a “prepare to render” message requesting thoseimages, and so on until the dependency tree has ended.

At each stage, the effect proxy checks whether the required image orrendered output is cached, and informs the render manager.

So, for example, if a prepare to render message is sent to a motiontracker specifying image 5, it might respond to say it requires arendered output (by itself) for image 4. The render manager then sends aprepare to render message to the motion tracker for image 4, and themotion tracker responds to indicate that it requires image 3 and so on.In this way a list of render jobs which are needed before the requiredimage (image 5) can be rendered is built up. Rendered outputs which areheld in the cache are not included on the render manager's job list.

The same thing happens where an effect requires the rendered output of apreceding effect, and so on down a chain of effects.

At the end of this process, the render manager sets all of the requiredjobs going, in a reverse order so that the currently required image isnot rendered until all of its dependent images are rendered.

As an optimisation, the render manager can detect from the graph whatthe inputs to each effect are. So, the effects server can send apredetermined code (e.g. a null reply) to say simply “all of the inputsto this effect are required”.

As a further extension, the same protocol can be used so that eacheffects server can notify the render manager if its output is the samebetween adjacent images. A simple example of this is a (fixed) parameterplug-in, where the output is invariant. A further example is any othereffect where the outputs have already been prepared and cached, so thata straightforward detection can be made as to whether successive outputsare identical. In response to such a notification, the render managerpass the information on to an effects server which is later in thedirected acyclic graph. That effects server can then (if appropriate)render only one of a range of images and repeat that output for otherimages where its input remains the same.

FIG. 6 schematically illustrates a graph editor window 600 and a palettewindow 610. These are displayed on the display screen 160 under thecontrol of the core 340.

The palette window 600 contains a number of icons 620, each mapped toand representing a different possible effect for which plug-ins exist onthe system. Using a mouse control, the user can “drag” these icons intoa scrollable graph window 610. The icons are arranged by the user in thegraph window with respect to one another and can then be linked up withlogical links 630, shown in the window as graphical lines.

The links 630 represent a passing of the output of an effect to theinput of a subsequent effect, and (in this embodiment) always have adirection from the bottom of the graph towards the top of the graphwindow. So, the example shown in FIG. 6 has an image loader icon 640passing its output to a lighting effect 650.

As the user sets up graphical links in the graph window, the core 340sets up logical links to determine the way in which rendered output ispassed from one effect plug-in to another.

The way in which the graphical links are created will now be describedwith reference to FIG. 7. The logical links will then be described withreference to FIG. 9.

In FIG. 7, the user has selected (e.g. with a mouse click) the lightingeffect and now has a moveable graphical line 720 directed from the icon650 towards a mouse pointer 730. As the mouse pointer approaches amixing effect icon 700, the mixing effect icon is magnified, orsurrounded by an enlarged 710, showing two input ports at the bottom ofthe border 740. As the mouse pointer approaches one of the input ports,the graphical line 720 snaps onto that input point and can be fixed inplace with a mouse click. This establishes a logical and graphicalconnection between the effect 650 and the effect 700.

Once these logical and graphical connections are established, the usercan “box” a linked group 800 of the effects icon in the graph window.Here, to “box” means to draw a box around the group in a standard wayusing a computer mouse. (One way that this is implemented is to clickand hold at the top left hand corner of the box, drag the mouse to thebottom right hand corner and then release the mouse button. It is astandard way of selecting or choosing plural screen objects).

The user is then able to drag the linked group of effects into thepalette area This creates a new, composite, effect icon 810 having a setof inputs formed from the inputs to the group and a set of outputsformed from the outputs of the group. In logical terms, instead of theeffect icon 810 being mapped to a particular plug-in, it is mapped to alinked group of plug-ins interconnected in a particular way.

The composite effect icon 810 then forms part of the palette for use bythe user in designing a graph. Later, if the user wishes to make use ofthe composite icon 810, he simply drags it into place on the graphwindow. Preferably, the effect icon 810 remains as a single icon on thegraph window, but in other implementations it can expand out into theoriginal group 800. As a further alternative, it can be displayed in itscompressed form as a single icon, but with an “expand” button displayedso that the user can click on the expand button to display the originalgroup of icons 800. In any event, the composite effect provided by theicon 810 is a copy of the original group 800.

FIG. 9 illustrates the data storage underlying this process. In FIG. 9an icon 850 has been dragged from the palette icons 620 in the palettearea 600 into the graph editor area 610.

Associated with the palette area 600, and stored in the palette 356shown in FIG. 3, is a data structure arranged as a tree having a root860 and individual data items 870 depending from that root. Each dataitem 870 represents one effects icon 620, except in the case of acomposite effect such as an effect 875. Here, the effects icons (3 a, 3b) forming that effect are arranged in a sub-structure depending fromthat data item 875.

A similar data structure exists to store effects in the graph editorarea Here, a root 880 is shown having just one effect 885 depending fromit. If a number of effects are grouped together in the graph editor areaand dragged to the palette, they form a further composite effectstructure similar to the structure 875.

FIG. 10 schematically illustrates a viewer window. The viewer windowcomprises an image display area 900, various “property pages” 910, andeffects control 920 (here shown as a positioning cross-wire in theexample of a lighting effect), and a “button bar” 930.

The basic layout of the viewer window is set by the core framework, andis standard from effect to effect. However, the particular items whichcan be adjusted using the property pages 910 are set by the effect UI320 corresponding to a particular effect. to The effect UI also providesdisplay details for the control 920.

So, in the example shown, the cross-wire 920 determines the source ortarget position of the light in the lighting effect The user can dragthe cross-wire using a computer mouse. Dragging the cross-wire changesthe parameter (x,y) values associated with that control, and so theprocedure of FIG. 4 (update parameter values) is initiated. As the lastpart of that procedure, in the step 408, the effect UI issues thecorrected parameter value to the viewer window. At that stage, thecross-wire is re-drawn in its new position. So, although it appears tothe user that the dragging action has moved the cross-wire to itsultimate position, in fact the dragging action created a parameterupdate which, by the route shown in FIG. 4, resulted in the movement ofthe cross-wire.

FIG. 11 schematically illustrates an initial arrangement of theoperating software. This represents the situation before any renderinghas to be carried out in a particular operating session of theapparatus.

The plug-ins are implemented under the Windows operating system as“dynamic load libraries” (DLLs). DLLs are generally large files whichcan contain program code, data and subroutine libraries. Conventionally,to conserve memory and improve system performance, a DLL loaded intomemory when first needed for execution or initiation of a particularprocess handled by that DLL. In the present embodiment, this idea ofconserving memory and improving system performance is taken one stepfurther.

So, when an effect icon is first taken from the palette area,conventionally the DLL corresponding to that effect would be loaded intomemory to provide the core 340 with sufficient information (e.g.interconnectivity with other effects icons) for the graph to be built.

In the present embodiment, the DLL for that effect is not loaded at thatstage. Instead, so called “metadata” 1000 representing that effect isloaded. The metadata provides the core with information defining theinterconnectivity of the effect with other effects (e.g. number ofinputs and outputs). This enables the core to build up a graph withoutthe need to load any DLLs, so saving memory by not loading large filesuntil they are absolutely needed.

If a viewer window is opened relating to an effect, or if the compositeeffect is executed by any other means, then the DLLs are loaded and themetadata discarded or ignored.

FIGS. 12 to 14 schematically illustrate features of the effects plug-inswhich (amongst other things) facilitate automation of the system.

FIG. 12 schematically illustrates a previously proposed effects plug-in.This effect takes image information (shown as a “clip” 1300) and acts onthe basis of three processing parameters PI, P2 and P3 (such as lightingpositions etc.). In the plug-in of FIG. 12, the parameter values are setwithin the plug-in, i.e. by bespoke program code written as part of theplug-in. This makes overall control of the parameters—e.g. for ananimation system where the parameters vary with time, or in anarrangement where a parameter such as a lighting position is varied byanother effect such as a motion tracker—very difficult, requiringadditional code within the effects plug-in and often multiple versionsof the effects plug-in.

FIG. 13 schematically illustrates another approach according to anembodiment of the invention. Here, each parameter is defined by aseparate plug-in 1320, linked to the “main” effects plug-in 1330 in thesame way as the links between effects are defined in the graph editordescribed above. In fact, the description given above was asimplification of the whole process, the simplification being made atthat stage to assist in the explanation.

The parameter plug-ins are normally hidden from the user, for example bydisplaying them at screen positions in the graph editor and palettewhich are “off the page”.

So, if an effect is to be operated in a self-contained, non-animatedmanner (i.e. without importing parameter values from other effects),then parameters are set for each parameter plug-in 1320 using the maineffect's viewer window.

If a parameter is to be defined by another effect's output, e.g. aposition value being provided by a motion tracking effect, then all thatis required if for the logical link between the main effect plug-in 1330and the appropriate parameter plug-in 1320 to be severed and a link tothe motion tracking effect initiated.

In order to understand how this system can assist in animation,reference is made to FIG. 14.

FIG. 14 shows the left-right split between core and plug-in shown firstin FIG. 3. On the left hand side of FIG. 14, a proxy effect (PE) 1340 isprovided for the “main” effect server 1350. Proxy effects 1360 are alsoprovided for each of the parameter plug-ins 1320. These proxy effects1360 are of a much simpler nature than the proxy effect 1340, andcommunication between the proxy effects 1360 and the parameter plug-ins1320 uses a simplified subset of the communication protocol between theproxy effect 1340 and the effects server 1350.

In actual fact, the proxy effects 1360 can be a single data value (in anon-animated system) or a list of values in an animated system. In ananimated system, the list of values can be expressed as “key frame”values, i.e. data values set for particular images in a sequence, withintervening values being interpolated by the core according to a linearor user-defined non-linear interpolation. So, animation can be set up ina particularly simple and convenient way without having to write bespokeanimation software within each plug-in.

Relating this description to that given earlier about dependenciesbetween effects, when a “prepare to render” message from the rendermanager is received by an effects server 1350, it can respond to saythat it requires all of its inputs before that output can be provided.Included in the effect's inputs are of course the parameter plug-ins, sothe next stage would be for the render manager to send a prepare torender message to each parameter plug-in. If the parameter plug-incontains a single value, or if the current image is a key frame, thenthe parameter plug-in is ready to provide the appropriate parameter atrender time. If, however, the parameter plug-in contains animation dataand the current image is not a key-frame, the parameter must first beinterpolated before it can be used in the effect.

FIG. 15 schematically illustrates a system cache 1100. This is aschematic view of the whole cache area—in fact, as described earlier,the cache can also be viewed as plural individual caches associated withrespective proxy effects, but since memory resources are dynamicallyallocated between such individual caches the representation of FIG. 15is also a valid one.

The cache is provided in the system memory 150 and is able to storeimages 1110 and non-image rendered outputs 1130 from effects (e.g. amotion vector in the case of a motion tracking effect).

The idea of the cache is to store the rendered output (whether or notthis is an image) of each effect in the directed acyclic graph. In thisway, if an effect is altered at a particular position in the directedacyclic graph, effects below that position do not need to be re-renderedto provide the new output. Instead, the cached outputs can be re-used. Afurther benefit is to assist in and speed up undo/redo operations, bystoring the output of particular effects (those with open viewerwindows) before and after a parameter change is made. The correspondingparameter change is also stored, so that the parameter change can beundone or redone simply by loading the appropriate material from thecache memory 1100. This is under the control of the undo/redo objectswritten by the effects servers when a change is made.

Images take up very much more memory space than a simple data value likea motion vector—perhaps a million times as much memory space. So, inthis embodiment, when the cache memory approaches its capacity andanother image is to be stored, the least recently accessed image in thecache is deleted to make room for the newly stored image. However, otherdata in the cache—parameter values, non-image rendered outputs and soon—is not deleted during an operating session of the apparatus becauseit consumes such a tiny amount of memory space in comparison with animage. This information is then available for re-use, or for anundo/redo operation, as long as the information remains valid throughoutan operating session.

In practice, it can be left that the plug-in specifies whether a dataitem is flushable or not, with image data items being set by conventionas flushable, and non-image items being set as non-flushable.

FIG. 16 schematically illustrates an asynchronous-synchronous converted1200 between the core 340 and an effect server.

The converter 1200 receives asynchronous re-rendering commands from theto render manager in the form of a “To do” queue, i.e. a list ofrendering jobs to be done. When a job is finished, a “finished” messageis returned from the converter 1200 to the render manager.

The converter 1200 receives the asynchronous job requests and issuessynchronous requests to the appropriate software plug-in. This meansthat the interface 1200 passes a control “thread” (a Windows term) tothe software plug-in, which retains control of the thread until the jobis complete. Only then, the software plug-in returns the thread to theinterface, which responds by issuing the “finished” message to the core.

At initialisation, the core interrogates each plug-in (or rather themetadata associated with that plug-in) to determine whether the plug-incan handle synchronous or asynchronous communication. If a hardwareplug-in (e.g. a peripheral card for rendering in a particular way) or anasynchronous software plug-in, possibly running on a different machine,is installed in place of a software plug-in, that plug-in interacts withthe core (in fact the render manager which initiates rendering tasksasynchronously) via the asynchronous interface, as hardware acceleratorsare much better suited to operation in this way. So, in this case, theconverter 1200 is bypassed.

The converter may be implemented as part of the core or part of eachrelevant plug-in.

Accordingly, by the counter-intuitive step of providing a converter 1200between two pieces of software, an efficient hardware interface isprovided for a later upgrade to dedicated hardware.

Although illustrative embodiments of the invention have been describedin detail herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various changes and modifications can be effectedtherein by one skilled in the art without departing from the scope andspirit of the invention as defined by the appended claims.

We claim:
 1. Video processing apparatus comprising: input means forinputting images of an input video sequence; processing means forprocessing the video sequence according to user adjustable video effectsparameters; generating means for generating output images of an outputvideo sequence; rendering means for rendering images of said outputvideo sequence according to a priority order of: (i) an image or imagescurrently displayed for view by a user; (ii) first and last images ofsaid output video sequence; and (iii) remaining images of said outputvideo sequence; and a display for displaying rendered images. 2.Apparatus according to claim 1, wherein said rendering means is operableas a background process within said apparatus.
 3. Apparatus according toclaim 1, further comprising: means for detecting a parameter change by auser which makes a current rendering operation invalid; and means,responsive to such a detected parameter change, for aborting a currentrendering operation by said rendering means and for restarting operationof said rendering means.
 4. Apparatus according to claim 1, furthercomprising user-operable sequence selection means for selecting a subsetof said input video sequence to be rendered to form said output videosequence.
 5. A video processing method comprising: inputting images ofan input video sequence; processing the video sequence according to useradjustable video effects parameters; generating output images of anoutput video; rendering images of said output video sequence accordingto a priority order of: (i) an image or images currently displayed forview by a user; (ii) first and last images of said output videosequence; and (iii) remaining images of said output video sequence; anddisplaying the rendered images.